The noise factor: irrelevant search results on the World Wide Web

Finding information on the World Wide Web is easy; finding relevant information is not. While search engines provide a more directed approach to resource discovery than browsing, the pages they identify as “matching” a query often have little relevance to the information being sought. To examine the relationship between query terms and the pages they match, the queries that were used to locate five Web pages were collected over a five-month period. The relevancy of page content to the query terms was then analyzed. Page content was judged as being irrelevant to more than one-third of the queries. Search engines disagreed; a test page link appeared within the first one percent of the total number of links retrieved in response to forty percent of those queries. This supported the supposition that conducting searches with popular search engines often results in too many links with little relevance to the query terms. An alternative approach is therefore proposed in which metadata, hyperlinks, and other subject-related HyperText Markup Language (HTML) tags are used to improve the effectiveness of Web queries. By relying on the structural components of HTML documents, it should be possible to conduct intelligent searches that yield more relevant results.